System and Method for Initiating Actions and Providing Feedback by Pointing at Object of Interest

ABSTRACT

A system and method as described for compiling feedback in command statements that relate to applications or services associated with spatial objects or features, pointing at such spatial object or feature order to identify the object of interest, and executing the command statements on a system server and attaching feedback information to their representation of this object or feature in a database of the system server.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.12/645,231, entitled “System and Method for Initiating Actions andProviding Feedback by Pointing at Object of Interest”, filed Dec. 22,2009, which claims the benefit under 35 U.S.C. §119(e) to U.S.Provisional Application No. 61/139,843, filed Dec. 22, 2008 entitled“System and Method for Providing Feedback by Pointing at Object ofInterest,” the entire contents of which are incorporated herein byreference.

FIELD OF INVENTION

The present invention relates generally to systems and methods forproviding feedback relating to spatial objects, elements, or functionalartifacts related to spatial objects, and controlling actions orservices that are performed in a functional dependency of the spatialobject. More specifically, the present invention relates to distributedsystems and methods that initiate or manipulate activities and services,bookmark, or attach comments, hyperlinks, ratings, or multimedia torepresentations of the spatial objects in database systems, by pointingat the spatial object or feature of interest in the real-world.

BACKGROUND OF THE INVENTION

Geotagging and Geoblogging are increasingly popular techniques forattaching location information to various types of media. Geotagging,sometimes also referred to as Geocoding, is the process of addinggeographical identification metadata to media such as websites, RSS(“Rich Site Summary”) feeds, or images, typically, in the form ofgeospatial metadata. However, there are some differences betweenGeotagging and Geocoding. Generally, Geotagging information associatedwith a location consists of latitude and longitude coordinates, whichalso may include altitude, bearing, and place name information.Geocoding, on the other hand, refers to the process of takingnon-coordinate based geographical identifiers, such as a postal address,and assigning geographic coordinates to them (or vice versa).

Geotagging media assists system users to find a wide variety oflocation-specific information. For example, a system user can findimages taken near a given location by entering latitude and longitudecoordinates into a Geotagging-enabled image search engine.Geotagging-enabled information services also can potentially be used tofind location-based news, websites, or other resources.

Geoblogging, in contrast to geotagging, attaches specific geographiclocation information to blog entries. The geographic locationinformation is typically identified by geotags, which usually consist oflatitude and longitude information. Searching a list of blogs andpictures tagged using geotag technology allows system users to selectareas of specific interest to them on interactive maps.

Similar to geotagging and geoblogging, the use of “pointing at”technology allows system users to point out spatial objects and featuresin order to retrieve information related to them stored in a database.This technology is becoming increasingly popular predominantly indomains, such as location-based services (“LBS”) or gaming.

The current generation of handheld devices, such as cellular phones andpersonal digital assistants (“PDAs”), include all of the components andsensors required to derive the system user's position and direction ofpointing, as well as the modules for wireless communication withinformation storing databases. These components, together with athree-dimensional representation of the environment in a database,enables the identification of spatial objects or features simply bypointing at them.

Generally, conventional pointing technology methods retrieve informationabout the object being pointed at; however, there is no two-waycommunication between the system user and spatial objects. Establishinga bi-directional communication channel between a system user and spatialobjects will allow interaction between the system user and theenvironment, and lead to more efficient and effective use of informationand services associated with spatial objects or features in theenvironment around the system user of a handheld or other device.

SUMMARY OF INVENTION

The system and method of the present invention is directed to compilingfeedback and/or command statements that relate to applications orservices associated with spatial objects or features, pointing atspatial objects or features in order to identify the object of interest,and executing a command statement on a system server or attaching thefeedback to the representation of the object or feature in a systemdatabase on a system server.

Feedback, according to the present invention, includes all forms ofsystem user input, including, but not limited to, hyperlinks, commentsand anecdotes, ratings, tags, keywords, images and sketches, audio orvoicenotes, and videos that relate to the spatial object or feature ofinterest. Command statements, according to the present invention,include all types of actions, as long as such actions relate toappliances, applications, or services on the system server used torelate information to and with the spatial object or feature ofinterest. Command statements include, but are not limited to,bookmarking objects of interest, launching applications or servicesrelated to the object or feature, or configuration statements thatrelate to the state of any application or appliance associated with theobject or feature. Applications as used herein, include but are notlimited to, information systems, database management systems, webapplications, dedicated and embedded applications, and any other toolthat functions and is operated by a computer. Appliances as used herein,include but are not limited to, automated teller machines, meters,automated systems for doors, lights and displays, household devices, andother electrical and mechanical instrument or apparatus designed or usedas a means to a specific end. And services as used herein, include butare not limited to, web services, bookmarking and registration services,electronic counter services, automated teller services, vending andticketing services, and any other software system designed to supportinteroperable machine-to-machine interaction over a network.

According to the present invention, an object of interest is identifiedby a system user by pointing at it. By doing this, the system user'sposition and direction of pointing are used to create a ray inthree-dimensional space, which is then intersected with atwo-dimensional projection of the system user's current scene. In thisscene, the object being pointed at is subsequently identified and usedas a dispatcher to associate functional units on the system server. Thetwo-dimensional projection of the current scene is generated from arepresentation of the environment stored in a database on the systemserver. This representation of objects or features that define theenvironment on the system server is linked to information, applications,services, appliances, and other functional units that relate toindividual objects or features.

The system client according to the present invention consists of a userinterface that allows the system user to select command statements orcompile feedback in any form, such as hyperlinks, ratings or comments,or multimedia. Feedback may be provided for any object, feature,service, or function as long as it is associated with correspondingobjects or features existing in geographic space. Examples of feedbackinclude comments about museums, photos of monuments, videos ofbuildings, bridges, roads, etc., but also audio tags for food or servicein a restaurant, notes of problems associated with some traffic element(hidden highway speed sign), or even personal comments about experiencesor happenings associated with other spatial objects or features.Feedback also may be tailored to suit specific sites or exhibitions thatare displayed only for a short period of time, but wish to collectfeedback about performance or preference of displayed items or artists.

Command statements may be presented for selection to a system user invarious forms, such as, for example, buttons on a user interface or menuitems. Command statements may be addressed directly to appliances,applications, or services linked with spatial objects or features in thereal-world through corresponding associations with the representationsof these spatial objects or features on the system server or in thesystem database. Examples of command statements of this type are (i)switching on and off the light of a building by simply pointing at thebuilding in the real-world, (ii) registering the real-world object witha social bookmarking service by pointing at the object in thereal-world, or (iii) starting, stopping, or pausing a virtual servicerunning on a system server, but associated with the representation ofthe spatial object or feature by pointing at the spatial object orfeature in the real-world.

A system client, typically a mobile device, is adapted for wirelesscommunication with a virtual system or database that contains spatialobjects and features representing the system user's environment andlinks to associated functional units. In order to identify the spatialobject or feature of interest on the system server, the system clientincludes modules for acquiring position information, e.g., GPS foroutdoor positioning or WLAN (“wireless local area network”)triangulation for indoor positioning, as well as directionalinformation, i.e., electronic compass, pitch, and yaw.

In operation, the system user points at the spatial object or feature ofinterest in order to collect current position and direction information.This information is then, together with the command statement or systemuser-generated feedback and contextual information, sent to the systemserver. At the system server, it is separated into appropriate segments.These segments are used to (i) identify the spatial object or feature ofinterest, (ii) identify the type of command or feedback, and (iii)dispatch the command to the associated execution unit or attach thefeedback to the representation of the spatial object or feature ofinterest stored in the system database.

The following are representative examples of the present invention:

As a first example, if Person A is having dinner in Restaurant C and thedinner is delicious, Person A may want to share the experience withothers. Currently, Person A's options are limited to (i) informing therestaurant owners about the great dining experience by orally tellingthem, or sending letter or e-mail, (ii) accessing the Internet at homeor somewhere else and posting a comment about the great diningexperience at Restaurant C, for example, by looking up the RestaurantC's webpage and posting a comment in the guestbook, or (iii) posting ablog about the experience.

The present invention enables system users, such as Person A, to standin front of the restaurant, like Restaurant C, type comments or a noteon their handheld device about the great dining experience, and point atRestaurant C in order to submit the comments or note to the systemserver. At the system server, the message containing the comments ornote is separated into pointing information from which the object isidentified, and the system user's comments or note are then attached tothe spatial object or feature of interest. The information about thespatial object or feature of interest is subsequently available foreverybody consulting the web or pointing at this object or feature inthe field in order to retrieve information about the object or feature.

A second example relates to real estate and home shopping. Person K isexploring a neighborhood in search of a new home. While exploring theneighborhood, Person K keeps notes on homes that are for sale. Some ofthe homes are more interesting than others and in order to keep notesand prepare for meetings with various real estate agents, Person Ktediously writes down the corresponding addresses, along with the phonenumbers of agents. Using the present invention, Person K would simplypoint at the building of interest and register it with a bookmarkingservice, without having to write down anything at all. Back at home,Person K could use the bookmarking service to generate a list ofaddresses for the bookmarked homes or a list of phone number of agentsthat are in charge of the bookmarked homes.

The present invention combines pointing, geotagging, and geoblogging inorder to define a novel system and method, which enables system users toadd information or media to a spatial object or feature in the database,rather than just attaching geographic location information to media. Assuch, the present invention differs from geotagging and geoblogging in anumber of ways, these include, but are not limited to, not attachinglocation information to specific media, but attaching the media to therepresentation of the spatial object or feature in the database simplyby pointing at the spatial object or feature in the real-world.Moreover, the present invention extends geotagging and geoblogging byallowing system users to address objects or features within the systemuser's field of view by pointing at the reference object or feature, andattaching information about this object or feature to its representationstored in a system database.

The present invention further extends social bookmarking services by amethod that allows bookmarking objects or features within the systemuser's field of view simply by pointing at the object or feature. Thepresent invention extends spatial interaction with real-world objects orfeatures by a pointing mechanism enabling interaction between systemusers and representations of real-world objects or features.

The present invention enables system users to execute commandsassociated with the representations of spatial objects or features on asystem server by pointing at their counterpart in the real-world. Thepresent invention extends remote control of services or appliances byallowing the control and management of appliances, applications, orservices by pointing at the object or feature in the real-world.

The present invention will be described in greater detail in theremainder of the specification referring to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a representative system for carrying outthe present invention.

FIG. 2 shows a representative flow diagram for carrying out the methodof the present invention.

FIG. 3 shows a representative user interface for a system client forproviding feedback about a specific spatial object or feature ofinterest.

FIG. 4 shows a representative diagram relating to HERE and THERElocations.

FIG. 5 shows a representative example of a visual scene which systemusers may use to provide feedback by pointing.

FIG. 6 shows a representative example of levels associated with spatialobjects and features to which system user-generated feedback may beattached.

FIGS. 7A and 7B show representative examples of pointing at differentelements of a building for attaching system user-generated feedbackinformation.

FIG. 8 shows a representative example of elements of a building to whichsystem user-generated feedback may be attached.

DETAILED DESCRIPTION OF THE INVENTION

The system and method of the present invention are for compilingfeedback or command statements that relate to applications or servicesassociated with spatial objects or features, pointing at spatial objectsor features in order to identify an object or feature of interest, andexecuting the command statement on a system server or attaching thefeedback to a representation of this object or feature in a systemdatabase on a system server.

FIG. 1, generally at 100, shows a general overview of the system of thepresent invention. FIG. 1 also shows the general information data flowamong the elements of the system of the present invention. Further, FIG.1 shows the basic workflow according to a method of the presentinvention, which will be described in greater detail subsequently.

Referring to FIG. 1, system 100 of the present invention is shown inrelation to the real-world 102. The real-world 102 represents a visualscene that would be viewed by the system user. System 104 includessystem client 106, system server 108, and storage and executionenvironment 110. As shown at 102, System User 124 will be located inenvironment 122 and will see visual scene 120 which is part ofenvironment 122. Visual scene 120 includes one or more spatial objects126 that may be pointed at by System User 124.

As stated, system 104 includes system client 106, system server 108, andstorage and execution environment 110. System User 124 interactsdirectly with system client 106 that includes User Interface andFeedback Editor 132, and Positioning and Pointing Modules 134. SystemUser 124 principally interacts with system client 106 at User Interfaceand Feedback Editor 132. When System User 124 points system client 106at spatial object 126, Positioning and Pointing Modules 132 candetermine the position of the specific spatial object or feature beingpointed at and supplies this information to User Interface and FeedbackEditor 132 for display to System User 124 that will relate to theidentification of the pointed at object or feature.

System server 108 receives mapped information relating to environment122 in which System User 124 is located. More specifically, systemserver 108 at 136 receives three-dimensional (3-D) representationinformation about environment 122 based on the location and pointingdirection of system client 106. The geographic location may be providedto system server 108, by, for example, a GPS sensor associated withsystem client 106. Other methods of determining environment scene may beused and still be within the scope of the present invention.

The three-dimensional representation of environment 122 at 136 isprovided to Scene Generator and Object Identification 140. UserInterface and Feedback Editor 132 sends system user requests foridentification of spatial objects or features of interest being pointedat by System User 124 to Command Parser and Feedback Processor 138.Command Parser and Feedback Processor 138 determines if the request is acommand statement only or whether there is a feedback statement as partof the request. Command Parser and Feedback Processor 138 transmitscommand statements to User Information and Interaction Engine 142 andfeedback statements to Scene Generator and Object Identification 140.

Scene Generator and Object Identification 140 provides the identity andlikelihood of the accurate identification of objects or features ofinterest to Feedback Linker and Command Dispatcher 144. The purpose ofFeedback Linker and Command Dispatcher 144 is to dispatch system usergenerated feedback or actions to the executing instance for objects orfeatures of interest in Storage and Execution Environment 110, and/or togenerate feedback for the separate system user in case of inaccurateobject or feature identification. Feedback Linker and Command Dispatcher144 sends system user-generated feedback to system database 146 forlinking with stored information relating to spatial objects or features,and commands to be executed by appliances, applications, and services at146. Feedback Linker and Command Dispatcher 144 also sends informationto Command Parser and Feedback Processor 138 and informs UserInformation and Interaction Engine 142 of the performance of the objectidentification process, and the success, failure, or state ofdispatching and linking system user generated feedback and commands tothe corresponding instance. User Information and Interaction Engine 142sends information about the outcome of object or feature identification,and/or dispatch and linking operations to User Interface and FeedbackEditor 132. Certain of this information will be displayed on the userinterface of User Interface and Feedback Editor 132. For example, UserInterface and Feedback Editor 132 may display the success or failure ofobject or feature identification, state of feedback linking, and/orprogress of command execution. Other information may be sent from UserInformation and Interaction Engine 142 to User Interface and FeedbackEditor 132 for the display and still be within the scope of the presentinvention.

As stated, Feedback Linker and Command Dispatcher 144 sends systemuser-generated commands and, if necessary, the object or featureidentity information to appliances, applications, and services 148. Byway of example only, appliances, applications, and services, include,but are not limited to, electronic-mechanical devices controlled withinthe scope of the object or feature of interest (e.g., automated dooropener), software applications associated with the object of interest(e.g., electronic meters), and dedicated or generic bookmarking orregistering services. Other appliances, applications, and services mayapply and still be within the scope of the present invention.

It is understood that the elements of system server 108 may be separatemodules or integrated into one or more modules and still be within thescope of the present invention. It is also understood that the elementsof client serve 106 may be separate modules or integrated into one ormore modules and still be within the scope of the present invention. Itis further understood that the elements of Storage and ExecutionEnvironment 110 may be separate modules or integrated into one or moremodules and still be within the scope of the present invention.

Referring to FIG. 2, shown generally at 200, the steps of the method ofthe present invention will be described. In describing the steps of thepresent invention according to FIG. 2, reference also will be made toFIGS. 1 and 3.

Generally, Step 1 at 202 is for inputting pointing information forobject or feature identification along with command statements orfeedback associated with a spatial object or feature of interest. Thisis performed by the system user selecting a command statement orgenerating feedback by entering specific information using userinterface (“UI”) of system client 106, such as in User Interface andFeedback Editor 132 (FIG. 1). The UI is adapted such that the systemuser can insert multiple or just specific types of commands or feedback.Examples of command statements are (i) “start,” “stop,” “pauseapplications or services,” (ii) “bookmark items,” or (iii) configurationstatements. The purpose of command statements with regard to appliances,applications, or services is to control, manipulate, and/or inform theseentities. The command statements that are provided are by way of exampleand not of limitation. It is understood that other command statementsmay be used and still be within the scope of the present invention.Examples of feedback include, but are not limited to, hyperlinks orURLs, ratings, comments, tags, keywords, or all types of media items, ora combination thereof, depending on the scope and purpose of thesituation.

Preferably, command statements may be selected by the system user fromdrop down lists or menu items placed on the UI, or typed by the systemuser on a console interface on system client 106. Feedback generation,however, is preferably carried out using an application-specific systemuser interface. An example of an application-specific system userinterface for feedback generation is shown in FIG. 3 generally at 300.

Referring to FIG. 3, a preferable application-specific system userinterface is shown at 300 for providing feedback about specific spatialobjects or features. Application-specific system user interface 300 willenable the system user to add different types of feedback. In FIG. 3,the interface titled “Feedback by Pointing” is shown at 302. Thisinterface is shown with four sections. The first is hyperlink section306; the second is rating section 308; the third is comment section 310;and the fourth is multimedia section 312. It is understood that theremay be more or less than four sections associated with the interface andstill be within the scope of the present invention.

At the bottom of the system user interface 300 is button 304 foruploading to system server 108, system user generated feedback relatingto the spatial object or feature being pointed at. The activation ofthis button will be described in greater detail subsequently.

Referring to hyperlinked section 306, the system user can enterhyperlinks with regard to the spatial object or feature being pointedat. For example, if the system user is reading the menu posted outsideof a restaurant where the system user has enjoyed a meal, system usercould enter the hyperlink of the restaurant on the menu in hyperlinkedsection 306. In FIG. 3, there are four spaces provided for enteringhyperlinks. It is understood that there may be a more or less and thanfour spaces provided for entering hyperlinks and still be within thescope of the present invention.

At rating section 308, the system user can enter ratings about an objector service based on his/her experience. It is noted that rating section308 provides spaces for rating an object or service with a “Rating A,”“Rating B,” or “Rating C.” It is understood that more than these threeratings may be used and still be within the scope of the presentinvention.

Comment section 310 is used by the system user to input any commentshe/she has relating to the spatial object or feature. The comments maybe confined to the area provided or unlimited with a scrolling screen tocapture the full extent of the system user's comments.

Multimedia section 312 is used by the system user to input multimediaassociated with the object of interest. This may include lists ofpictures, videos, voice notes, or other multimedia sources ofinformation that may be available that relate to the object or feature.In order for the system user to add pictures, he/she will activatebutton 314 for picture download. Similarly, the system user wouldactivate button 316 for adding videos and button 318 for adding voicenotes. If there are other types of multimedia source information to beadded via the interface, there would be a button made available oninterface for that purpose. Although only three buttons for addingmultimedia information are shown for adding multimedia information,namely buttons 314, 316, and 318, it is understood that more or lessthan three buttons for adding multimedia information may be available oninterface and still be within the scope of the present invention.

After system user has entered the feedback information about the spatialobject using interface 300, and wants to link it to a representation ofthe spatial object or feature in the system server database, he/shepoints at the object or feature of interest and activates point@itbutton 304. This will upload the feedback information to system server108. It is also understood that the system user can point the devicehaving application-specific system user interface 300 at the object ofinterest before compiling the feedback data and it will still be in thescope of the present invention.

Referring to FIG. 1, when point@it button 304 is activated, systemclient 106 connects to Positioning and Pointing Module 134 and extractssystem client 106's (i) position in latitude, longitude, altitude, oraccording to x, y, and z positioning in an x, y, and z coordinatesystem, and (ii) direction in yaw and pitch. These measurements togetherwith the system user feedback or command statement are then submitted asa “request” to the system server for further processing. The requestthat includes the system user generated feedback or command statementand is sent from system client 106 to system server 108. Preferably, therequest will include at least the following components that are in BNF(“Backus Naur Form”) notation:

<Request> ::= <Context> (<Feedback> | <Command_Statement>)(<HERE_Location> | <THERE_Location>) <Context> ::= <Personal><Infrastructure> <Environmental> <Temporal> <Personal> ::= ‘User ID,Name, Age, Gender, Interests, etc.’ <Infrastructure> :: = ‘Type ofnetwork, bandwidth, etc.’ <Environmental> :: = ‘Weather, Noise, othersensory input’ <Temporal> ::= ‘Date, Time, User's path to this location,etc.’ <Feedback> ::= <URI> [<URI>] | <Comment> [<Comment>] | <Tag>[<Tag>] | <Keyword> [<Keyword>] | <Rating> [<Rating>] | <Multimedia>[<Multimedia>] <URI> ::= ‘any URI’ <Comment> ::= ‘any Comment orapplication-specific scheme-based Comment’ <Rating> ::= ‘any Rating orapplication-specific scheme-based Rating’ <Tag> ::= ‘any Tag that isassociated with the object’ <Keyword> ::= ‘any Keyword that isassociated with the object’ <Multimedia> ::= ‘any list of multimediaitems or files’ <Command_Statement> ::= <Command > [<Option> [<Option>]][<Command > [<Option> [<Option>]]] <Target> [<Target>] <Command> ::=‘any command associated with object, e.g., start, stop, pause,configure’ <Option> ::= ‘any option associated with command, e.g., timestamp’ <Target> ::= ‘target appliance, application, or service, e.g.,bookmarking service’ <HERE Location> ::= ‘user's current positionidentifying feature of interest’ <THERE Location> ::= ‘pointing data foridentification of object of interest’

Referring to the feedback statement shown above, there are threeprincipal components of the statement. These components are the (i)Context, (ii) Feedback or Command Statement, and (iii) HERE or THERELocation.

The Context component sets forth a set of properties that describe theidentity and interests of the system user, as well as the personal,infrastructure, environmental, and temporal characteristics under whichthe feedback statement was compiled and submitted to system server 108.

The Feedback component of the feedback statement consists of any URL,comment, rating, document, multimedia, or other piece of informationthat the system user wants to attach to the object of interest.Preferably, the Command Statement component is a three-valued statementthat includes information about what target appliance, application, orservice that needs to be manipulated when pointing at an object orfeature, and related options. In referring to related options, it isunderstood to mean optional commands for refining parameters orattributes, such as additional command attributes (e.g., user name),control parameters (e.g., safe execution only), or time-dependentexecution (e.g., now or a specific date/time).

The HERE or THERE Location component includes information that isrequired for identifying an object or feature of interest. A HERELocation refers to a spatial object or feature located at the systemuser's current location. A THERE Location refers to a spatial object orfeature that is located at a distant position. This is shown graphicallyin FIG. 4.

Referring to FIG. 4, generally at 400, HERE and THERE Locations will bedescribed in detail. System User 402 is standing at system user position406. This position would have a latitude, longitude, and altitude. Apredetermined distance around a system user's position defines HERELocation 404. If a spatial object, such as stop sign 408, is located atthe HERE Location 404, it could be identified by the system server as anobject at the HERE Location. It is understood that the definition of thesize of the HERE Location may be larger or smaller and cover a shapethat is different from what is shown in FIG. 4 and still be within thescope of the present invention.

Again referring to FIG. 4, if system user 402 desires to identify adistant object or feature of interest, such as building 414, system user402 at HERE Location 404 would point system client 106 in direction 410toward building 414. Distant building 414 would be located at THERELocation 412.

Again referring to FIG. 2, Step 2 at 204 provides for the generation ofa two-dimensional (2-D) representation of the system user's visual scenefrom the three-dimensional (3-D) model of the environment surroundingthe system user. In contrast to HERE Location 404 in FIG. 4, THERELocation 412 uses a different method for identification. The method bywhich distant objects of interest are identified is to generate a 2-Ddigital scene from the representation of the environment stored in thesystem database 146 of storage and execution environment 110 (FIG. 1).This digital scene will represent the system user's current field ofview. Within the scene, the system and method of the present inventionwill intersect a ray defined by the system user's current position andpointing direction with the spatial object or feature within the scene.This will result in the identification of objects or features ofinterest.

Referring to FIG. 2, Step 3 at 206 of the present invention will bedescribed. At Step 3, the object or feature of interest is identified.If, for example, there is a single object spatially dispersed from otherobjects to which system client 106 is being pointed, then the 2-D scenethat is generated only shows that object or feature, and it can becorrectly identified from information stored in the system database.However, to the extent that the identification of any spatial object orfeature of interest cannot be carried out with a sufficient level ofaccuracy and confidence, the system user is informed by the system ofthis, and asked to resubmit the request or choose the select object froma list of possible objects. This method identification will be describedin more detail referring to FIG. 5.

Referring to FIG. 5, generally at 500, the description for identifyingan object or feature of interest where there may be more than one objectin the pointing direction of system client 106 will be provided.Pointing system 502 is based on x, y, z coordinate system 504. Based oncoordinate system 504, system user 506 will have a latitude, longitude,and altitude. By pointing system client 106 in a specific direction, yawand pitch associated with system user's position will be generated. Asshown in FIG. 5, by the system user pointing in direction 516, there isgeneration of 2-D visual scene 508. In that visual scene, are building510 and silo 512. As shown in real-world representation 518, building510 and silo 512 are different objects that are spaced apart but in 2-Dvisual scene 508 they are overlapping. Without any additionalinformation, it could not be determined with any confidence whethersystem user 506 is pointing at building 510 or silo 512.

According to the method of the present invention, object or featureidentification is carried out using statistical methods and errorpropagation principles in order to assess the reliability of pointing.Examples of the statistical methods and error propagation principles arethe following t-test for correspondence of pointing direction and likelytarget objects or features, and—geodetic methods for assessingpropagation of GPS and compass errors in a visual scene. There also maybe default reliability criteria set for the system which if not met willnotify the system user of same and require additional information beforeidentification can be made and confirmed.

Referring to FIG. 2, at decision point 208, the system and method of thepresent invention determines whether or not identification of the objectof interest has been made with sufficient accuracy according to systemsettings. If the answer is “No,” then the system and method will move toStep 7 at 220 and inform the system user that identification could notbe made without additional information. If the answer is “Yes,” then thesystem and method will proceed to Step 4 at 210.

Referring to Step 4 at 210 in FIG. 2, preparation of context informationwill be described. As discussed above with respect to feedback data thatis provided by the system user using system client 106, there are threemain components of the feedback statement submitted to the systemserver; one of them is Context (contextual) information.

When contextual information is input by the system user into systemclient 106, it is processed to be added as part of the command statementor feedback statement submitted to system server 108. The accuratecapture of the Context information is important to ensure the quality ofthe data attached to a spatial object or feature. For example, if systemuser A and system user B are rating the “fear factor” they bothexperienced in crossing a park, the context of their crossing will haveimpact on their rating of the park. If system user A crosses the parkduring daylight and rates the park as open and friendly, that ratingwill reflect the time of day of the crossing. However, if system user Bacross the park at night and rates the park as dark and scary due tomany missing lights, this context is very different than the oneexperienced by system user A. Therefore, if the contextual informationof each crossing is attached to the rating, the differences in theseratings could clearly be understood and explained. As such, thecontextual information stored with each feedback item provides insightsabout the circumstances under which the system user feedback wasgenerated.

Referring again to FIG. 2, Step 4 at 210 provides examples of personalcontextual data that may be included. This information may includesystem user data such as the system user's name, age, gender, or otherpersonal information. Infrastructure contextual data may be the type ofsystem client device being used, such as a phone, PDA, or other device.The environmental contextual data may include weather conditions orother environmental conditions or factors. Temporal contractual data mayinclude the time of day. The above examples are not meant to be oflimitation and it is understood that other information may be providedand still be within the scope of the present invention.

At decision point 212, the present invention determines whether or notwhat is submitted to system server 108 is a command statement or afeedback statement. This will be carried out by the system server atCommand Parser and Feedback Processor 138. If it is determined atdecision point 212 that what is received by system server 108 fromsystem client 106 is a feedback statement, then the present inventionwill proceed to Step 5 at 214. However, if what is received by systemserver 108 from system client 106 is a command statement, the presentinvention will proceed to Step 6 at 216.

Before discussing Step 5 at 214 in greater detail, generally, Step 5 isdirected to linking the individual feedback information generated by thesystem user to the spatial object or feature of interest identified atStep 3 at 206. These spatial objects or features may be any element thatis perceived, visually or otherwise, by the system user and mapped intothe system database. These spatial objects or features will also beidentified by means of a HERE Location, the system user's currentlocation, or a THERE Location, objects or feature at which the systemuser is pointing. Spatial objects or features of this type include, butare not limited to, rural objects, urban objects, and even indoorobjects. By way of example, urban objects include, but are not limitedto, water objects like lakes or ponds; generic city objects likebuildings or bridges; city features like crosswalks or traffic lights;or city object groups like districts.

An ontology of geographic space that may serve as a base for thefeedback system is represented in the class diagram shown in FIG. 6,generally at 600. This ontology of geographic information is meant toserve only as an example, and is understood to be an incompleteontology. It is further understood that information may be added to thisontology to make it complete and it would be within the scope of thepresent invention.

Referring to FIG. 6 at 602, the component information associated withany spatial object or feature is shown in what is referred to as“Legend” 601. These components are the contents of the system databaseat 604 and the system user-generated feedback at 606. These twocomponents will be used to find and define a spatial object or feature.In order to obtain the information set forth in Legend 601, it isnecessary to link the information that is accumulated at 606, which isshown as

$\frac{\langle{\langle{Feature}\rangle}\rangle}{spatialObject}.$

Class instances include different class levels to which feedbackinformation attaches. For example, one class level would be the featureclass, that includes ReliefFeature 634, Site 642, WaterObject 636,CityObjectGroup 644, GenericCityObject 638, TransportationObject 646,and CityFeature 640. This feature class would inherit all the propertiesof spatialObject 606. For purpose of example, referring to FIG. 6, anurban object will be described which is intended to be a feature ofinterest.

At a first class level, an object may be rural object 622, urban object624, indoor object 626, or other object 628. Rural object 622 may befurther defined by other subfeatures such as shown in phantom at 630.Similarly, indoor objects 626 may be further defined by the subfeaturesshown in phantom at 632.

Urban object 624 is shown with possible subfeatures. At this second orsub-level, the possible defining subfeatures are relief feature 634,site feature 642, water object 636, city object group 644, generic cityobject 638, transportation object 646, or city feature 640. Urban object624 would include at least one of these features. For purpose ofexample, the feature of interest is a particular art museum, and thefeature could be a general city object such as 638. Information aboutthis museum that is stored in a system database will be transmitted to608 where information relating to the 3-D geometric features and a typeand function of the museum would be provided that relate to thisfeature.

Next, user feedback information is generated at 610 based on system userinformation at 612, comments provided by system users at 614, ratinginformation about the museum at 618, media information, such as images,audio, or video information, at 620, or other information at 616. Thissystem user feedback is attached to spatial object at 606. Thecombination of this information at 606 and 608 will generate Legend 601that includes database content 602 and a system user generated input604.

According to the foregoing, the feedback statement generated at 610 bythe system user based on system user input is attached to the spatialobject information based on the level of granularity of the featureclasses. The level granularity of spatial objects or features may bebased on system settings or determinations by the system user. As anexample, system settings may provide that the finest granularity forspatial objects is the level of building blocks, so that when a systemuser points at some ornament that is part of the building, it willresult in attaching the feedback to the building. On the other hand, ifthe system settings are such that the ornaments of a building define thehighest granularity level, then feedback will be attached to an ornamentrather than to the building block itself. Therefore, the defined classlevel will determine where information will attach. This will beexplained in greater detail referring to FIGS. 7 and 8.

Referring to FIG. 7A, generally at 700, it shows at first representation702 of a scene where the pointing location is the building. Referring toFIG. 7B, generally at 701, it shows a second representation 703 of thesame scene where the pointing location is the front door of thebuilding. Referring to FIGS. 7A and 7B, they each show a 2-Drepresentation of the scene that includes building 704 and silo 706. InFIG. 7A, pointing location 708 is building 704 or the front of thebuilding 704. In this case, the class level of granularity would bebuilding blocks or the level right below building blocks (the front ofbuilding 704). As such, the feedback statement information would attachto building 704 or the front of building 704. In FIG. 7B, the pointinglocation is front door 712. This will mean that the level of granularitywould be ornaments of building 704 or features of the front of building704. Therefore, the feedback statement information would attach to frontdoor 712 as a feature of the building or the front of the building.

Referring to FIG. 8, generally at 800, features of building 704 to whichfeedback information may attach are shown. For example, feedbackinformation may attach to (i) Class Level 1: the solid form of thebuilding at 802, (ii) Class Level 2: façade 1 at 804 or façade 2 at 806,or (iii) Class Level 3: window 808 on façade 2 or front door 712 onfaçade 2.

If at decision 212, it is determined that the context information thatis provided from Step 4 at 210 is a command statement without a feedbackstatement, the present invention will proceed to Step 6 at 216. Thepurpose of Step 6 is to parse and execute command statements.

As set forth above, command statements are processed by system server104 at 138 and command statement information is transmitted to UserInformation and Interaction Engine 142 for processing. User Informationand Interaction Engine 142 processes the command information and sendsinstructions to User Interface and Feedback Editor 132 of system client106 (FIG. 2). The command information that is parsed from the commandstatements and sent may include, but not be limited to, “start,” “stop,”“pause applications or services,” “bookmark items,” or configurationstatements. These statements are launched from User Information andInteraction Engine 142 for execution by Appliances, Applications, andServices 146 associated with the object or feature of interest usingFeedback Linker and Command Dispatcher 144 as the intermediary betweenthe system user, object of interest, and executing entity.

In more detail, according to Step 6, an incoming command statement isparsed and associated with the spatial object or feature of interest inorder to engage an associated appliance, application, or service. Thisis carried out by a synchronized process involving Command Parser 136,Scene Generator and Object Identification 140, User Information andInteraction Engine 142, and Feedback Linker and Command Dispatcher 144.

In parsing the command statement, it is parsed into the core command,the options that accompanying this command, and the target appliance,application, or service at 148 addressed by the command statement. Thisinformation is transmitted to User Information and Interaction Engine142 which forwards the parsed command to Feedback Linker and CommandDispatcher 144. This information is then forwarded to appliances,applications, and services in Storage and Execution Environment 110 forexecution (FIG. 1). The execution engine at 148 will then engage thetarget application or service associated with the object or feature ofinterest and this causes these applications or services to behave asrequested by the system user. The following is an example of theoperation of the system of the present invention with respect to commandparsing and execution.

The following example will be directed to a pointing-based dialingservice for real estate agency. Conventionally, when exploring aneighborhood for a new home, a system user would have to find thetelephone number of the agent that manages the house of interest anddial the number in order to call the agent. In the case of apointing-based dialing service, the system user would simply point thesystem client device, such as a cell phone, at the real estate ofinterest, and hit the “Dial by pointing” button on the screen. Thecommand statement, for example, would include a core command, e.g.,“Start,” the target application, e.g., “Call Agent,” and the referencedobject, which provides the execution engine at 148 enough information toautomatically launch the call at 142. Once a call is launched, controlis handed back to the system user for further handling using the userinterface at 132.

According to the method of the present invention, once there isprocessing according to either Step 5 at 214 or Step 6 at 216, themethod proceeds to Step 7 at 220. At Step 7, there is generation ofsystem feedback and user information. The system feedback is feedbackinformation that is generated based on a system user's input andintended to inform the system user of the success or failure at Steps 3at 206, 4 at 210, 5 at 214, and 6 at 216.

When the object or feature of interest is identified at Step 3 withsufficient reliability, the command is executed according to Step 6 orthe feedback is linked to the spatial object or feature according toStep 5, and the system user is notified about the success of theoperation at Step 7. In situations when the spatial object of interestcannot be identified with sufficient reliability, system bypasses Steps4, 5, and 6 and the system user is notified of the identificationfailure. In situations where there is a failure, the system server isprovided with a list of likely spatial objects or features. Theseobjects or features are displayed on a system client 106, which providesthe system user with an opportunity to select the correct spatial objector feature and resubmit the request. The present invention also providesan additional option through system client 106. This option is for thesystem user to point at the object or feature of interest a second timeand resubmit the request.

The feedback system may also be adapted to be used for educationalpurposes either outdoors or indoors. Typically, educational systems takeuser input, such as the result of a multiple choice test or other typeof results, assess this feedback with respect to the correct answers,and provide appropriate feedback to the system user. The presentinvention is contemplated to be adapted to incorporate these types ofeducational systems. More specifically, the present invention may beconfigured such that applications which require testing system userfeedback against predetermined values may be implemented. As an example,a professor may decide to take his/her class on a field trip to ahistoric site or museum. Instead of going back to the classroom toexamine the students, he/she may want to test their knowledge in thefield. The feedback system of the present invention can implement thistype of educational activity and provides all of the components requiredto set up the system for this specific purpose.

It is understood that the elements of the systems of the presentinvention may be connected electronically by wired or wirelessconnections and still be within the scope of the present invention.

The embodiments or portions thereof of the system and method of thepresent invention may be implemented in computer hardware, firmware,and/or computer programs executing on programmable computers or serversthat each includes a processor and a storage medium readable by theprocessor (including volatile and non-volatile memory and/or storageelements). Any computer program may be implemented in a high-levelprocedural or object-oriented programming language to communicate withinand outside of computer-based systems.

Any computer program may be stored on an article of manufacture, such asa storage medium (e.g., CD-ROM, hard disk, or magnetic diskette) ordevice (e.g., computer peripheral), that is readable by a general orspecial purpose programmable computer for configuring and operating thecomputer when the storage medium or device is read by the computer toperform the functions of the embodiments. The embodiments or portionsthereof, may also be implemented as a machine-readable storage medium,configured with a computer program, where, upon execution, instructionsin the computer program cause a machine to operate to perform thefunctions of the embodiments described above.

The embodiments or portions thereof, of the system and method of thepresent invention described above may be used in a variety ofapplications. Although the embodiments, or portions thereof, are notlimited in this respect, the embodiments, or portions thereof, may beimplemented with memory devices in microcontrollers, general purposemicroprocessors, digital signal processors (DSPs), reducedinstruction-set computing (RISC), and complex instruction-set computing(CISC), among other electronic components. Moreover, the embodiments, orportions thereof, described above may also be implemented usingintegrated circuit blocks referred to as main memory, cache memory, orother types of memory that store electronic instructions to be executedby a microprocessor or store data that may be used in arithmeticoperations.

The descriptions are applicable in any computing or processingenvironment. The embodiments, or portions thereof, may be implemented inhardware, software, or a combination of the two. For example, theembodiments, or portions thereof, may be implemented using circuitry,such as one or more of programmable logic (e.g., an ASIC), logic gates,a processor, and a memory.

Various modifications to the disclosed embodiments will be apparent tothose skilled in the art, and the general principals set forth below maybe applied to other embodiments and applications. Thus, the presentinvention is not intended to be limited to the embodiments shown ordescribed herein.

1. A computer-implemented method for identifying an object in areal-world scene by pointing at the object with an electronic pointingdevice and electronically linking information to the object in a digitalrepresentation of the object stored in system storage, comprising thesteps of: (A) mapping a three-dimensional representation of a real-worldscene containing at least one object and transmitting thethree-dimensional representation of the scene to a system server; (B)generating a two-dimensional representation of the three-dimensionalrepresentation with the system server; (C) pointing the pointing deviceat the at least one object and collecting a current geodetic positionand pointing direction for the pointing device and creating a ray thatintersects with a two-dimensional representation according to thegeodetic position and pointing direction of the of the pointing device;(D) selecting at the pointing device command statements and compilingfeedback associated with the at least one object for identifying the atleast one object; (F) transmitting from the pointing device to thesystem server at least three segments of information, with the threesegments including (1) the geodetic position and pointing direction ofthe pointing device, (2) command statements, and (3) compiled feedback;(G) the system server separating the segments and (1) using the currentposition and pointing direction of the pointing device to identify theat least one object, (2) identifying the types of commands and compiledfeedback, and (3) dispatching the command statements for execution andattaching the compiled feedback to the at least one object; (H)identifying the at least one object of interest at the system server andtransmitting the identity of the object of interest to the pointingdevice, with the command statements being executed respect to the atleast one object of interest, and the compiled feedback being attachedto the at least one object of interest.
 2. The method as recited inclaim 1, wherein command statements include actions that relates toappliances, applications, or services on the system server relateinformation to the at least one object.
 3. The method as recited inclaim 2, wherein appliances include at least automated teller machines,meters, automated door systems, lights, displays, devices, orelectrical/mechanical instruments.
 4. The method as recited in claim 2,wherein applications include at least information systems, databasemanagement systems, web applications, dedicated and embeddedapplications, or any tool that functions and is operated by a computer.5. The method as recited in claim 2, wherein services include webservices, bookmarking and registration services, electronic counterservices, automated teller services, vending and ticketing services, orany software system for supporting interoperable machine-to-machineinteraction over a network.
 6. A computer-implemented method foridentifying an object in a real-world scene by pointing at the objectwith an electronic pointing device and electronically linkinginformation to the object in a digital representation of the objectstored in system storage, comprising the steps of: (A) inputting fromthe pointing device to a system server a geodetic position and pointingdirection of the pointing device with command statements generated bythe system user on the pointing device and feedback associated with theat least one object of interest; (B) mapping a three-dimensionalrepresentation of a real-world scene containing at least the one objectand transmitting the three-dimensional representation of the scene to asystem server; (C) generating a two-dimensional representation of thethree-dimensional representation with the system server; (D) identifyingwith the system server the at least one object and transmitting anidentity of the at least one object to the pointing device and if apredetermined level of accuracy in identifying the at least one objectof interest is not achieved, then step (A), otherwise step (E); (E)inputting from the pointing device to the system server contextualinformation; (F) determining if what was input at step (A) was a commandstatement or feedback and if what was input at step (A) was a commandstatement then step (G), otherwise step (H); (G) the system serverparsing and executing the command statement with respect to the at leastone object; and (H) linking the feedback to the at least one object andstoring the feedback in the system database with respect to the at leastone object.
 7. The method as recited in claim 6, wherein commandstatements include actions that relates to appliances, applications, orservices on the system server relate information to the at least oneobject.
 8. The method as recited in claim 7, wherein appliances includeat least automated teller machines, meters, automated door systems,lights, displays, devices, or electrical/mechanical instruments.
 9. Themethod as recited in claim 7, wherein applications include at leastinformation systems, database management systems, web applications,dedicated and embedded applications, or any tool that functions and isoperated by a computer.
 10. The method as recited in claim 7, whereinservices include web services, bookmarking and registration services,electronic counter services, automated teller services, vending andticketing services, or any software system for supporting interoperablemachine-to-machine interaction over a network.